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Abstract 


The  End-To-End  (ETE)  Test,  conducted  under  the  auspices  of  the  Department  of  Defense  Joint  Advanced 
Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E),  developed  a  synthetic  test  environment 
that  can  be  used  for  future  operational  testing  and  doctrinal  development.  This  synthetic  test  environment 
was  used  initially  to  conduct  developmental  and  operational  testing  of  the  Joint  Surveillance  Target 
Attack  Radar  System  (Joint  STARS).  As  designed  and  built,  it  may  be  used  to  conduct  future  testing  of 
systems  such  as  the  common  ground  station  (CGS),  the  All  Source  Analysis  System  (ASAS),  and  the 
Block  II  Army  Tactical  Missile  System. 

This  paper  will  describe  the  results  of  our  live  flight  operational  test  utilizing  satellite  communications  to 
distribute  the  synthetic  environment  and  lessons  learned  to  date.  Additionally,  the  experience  gained  in 
verifying  and  validating  a  distributed  synthetic  test  environment  using  the  Department  of  Defense 
Verification,  Validation  and  Accreditation  (VV&A)  Recommended  Practices  Guide  will  be  discussed. 

Background 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was  chartered  by 
the  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology),  Deputy  Director,  Test, 
Systems  Engineering  and  Evaluation  (Test  and  Evaluation)  “...to  investigate  the  utility  of  advanced 
distributed  simulation  (ADS)  for  both  developmental  test  and  evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E).  JADS  will  investigate  the  present  utility  of  ADS,  including  distributed  interactive 
simulation,  for  T&E;  and  finally,  identify  the  requirements  that  must  be  introduced  in  ADS  systems  if 
they  are  to  support  a  more  complete  T&E  capability  in  the  future.” 

The  approach  taken  by  JADS  was  to  select  discrete,  well-defined,  slices  of  the  broad  T&E  spectrum, 
conduct  ADS-augmented  test  activities  upon  specific  systems  that  are  representative  of  these  slices,  and 
address  the  issues  identified  within  our  charter.  In  addition,  ADS  activity  external  to  JADS  was  evaluated 
to  provide  as  broad  a  base  for  our  conclusions  as  was  possible. 

The  emphasis  of  the  JADS  JT&E  was  on  the  performance  of  the  ADS  components  and  their  contribution 
to  testing,  rather  than  any  particular  system  under  test  or  class  of  weapon  system.  Areas  of  interest 
included  network  capabilities  and  performance,  relationships  between  data  latencies,  and  ADS-induced 
data  anomalies. 

The  End-To-End  (ETE)  Test  was  one  of  three  tests  within  the  JADS  JT&E  program.  The  JADS  ETE  Test 
was  designed  to  evaluate  the  utility  of  ADS  to  support  the  testing  of  a  command,  control,  communications, 
computers,  intelligence,  surveillance  and  reconnaissance  (C4ISR)  system,  the  Joint  Surveillance  Target 
Attack  Radar  System  (STARS).  Joint  STARS  is  composed  of  both  an  airborne  and  ground  segment  along 
with  the  necessary  communications  subsystems.  The  E-8  airborne  system  and  the  ground  station  modules 
(GSM)  or  common  ground  stations  (CGS)  together  provide  the  surveillance,  target  detection,  and  tracking 
required  to  assist  commanders  in  understanding  the  enemy  situation  and  taking  action  to  destroy  enemy 


forces.  The  test  design  placed  Joint  STARS  and  the  appropriate  Army  targeting  and  attack  systems  in  the 
end-to-end  loop  of  detect,  track,  target,  cue  a  weapons  system,  and  assess  battle  damage. 

End-To-End  Test  Overview 


A  previous  shortfall  in  the  testing  of  C4ISR  systems,  especially  those  using  large  area  sensor  systems  such 
as  Joint  STARS,  was  the  inadequate  numbers  of  forces,  either  friendly  or  adversary,  available  to 
realistically  portray  the  expected  operational  environment.  In  addition,  systems  were  often  tested  in 
isolation  without  the  complementary  suite  of  other  command,  control,  communications,  computers  and 
intelligence  (C4I)  and  weapon  systems  with  which  to  interact.  The  original  Joint  Surveillance  Target 
Attack  Radar  System  (Joint  STARS)  Multiservice  Operational  Test  and  Evaluation  (MOT&E)  Plan,  dated 
21  February  1995,  identified  the  ability  to  test  the  requirement  for  Joint  STARS  to  provide  support  to  a 
notional  corps  area  of  interest  as  a  limitation  of  the  test.  As  stated  in  the  plan,  the  testing  of  this 
requirement  “would  require  both  a  completely  equipped  corps  operation/exercise  and  multiple  E-8 
aircraft.” 

Obviously,  the  requirement  for  a  completely  equipped  corps  operation/exercise  was  impossible  to  meet  in 
1995  and  is  even  more  impossible  now.  Plus,  even  if  the  testers  could  find  a  fully  deployed  corps  involved 
in  an  operational  mission,  as  was  roughly  the  case  in  Bosnia,  there  would  still  be  difficulties  in  testing 
how  well  Joint  STARS  supported  the  execution  of  attacks  against  detected  targets. 

This  test  limitation  was  the  major  basis  for  the  development  of  the  JADS  ETE  Test.  The  JADS  ETE  Test 
was  designed  to  add  an  augmented  operational  environment,  approximating  a  notional  corps  area  of 
interest,  and  a  complementary  suite  of  C4ISR  and  weapons  systems  with  which  Joint  STARS  would 
interact. 

The  ETE  Test  used  distributed  interactive  simulation  (DIS)  to  assemble  a  synthetic  environment  to 
evaluate  the  utility  of  ADS  to  support  the  testing  of  Joint  STARS.  The  intent  was  to  provide  a  complete, 
robust  set  of  interfaces  from  sensor  to  weapon  system,  including  the  additional  intermediate  nodes  that 
would  be  found  in  a  tactical  engagement. 

The  test  concept  was  to  use  ADS  to  supplement  the  operational  environment  experienced  by  the  E-8C  and 
light  ground  station  module  (LGSM)  operators  by  adding  additional  entities  to  the  battlefield  seen  by  Joint 
STARS.  Also,  by  adding  additional  elements  of  the  C4I  systems  that  Joint  STARS  interacts  with  and  a 
weapon  system,  the  Army  Tactical  Missile  System  (ATACMS),  to  engage  targets,  the  test  team  could 
evaluate  the  complete  battlefield  environment  from  target  detection  to  target  assignment  and  engagement. 
It  also  allowed  the  tester  to  evaluate  the  entire  thread,  or  the  individual  contribution  of  any  of  the  parts, 
and  to  evaluate  what  effects  an  operationally  realistic  environment  had  on  the  system  under  test. 

This  was  accomplished  by  adding  approximately  ten  thousand  simulated  targets  to  the  live  targets  seen  by 
the  radar  on  board  the  E-8C  aircraft.  This  presented  a  battle  array  approximating  the  major  systems 
present  in  the  notional  corps  area  of  interest  to  the  surveillance  operators  both  in  the  air  and  on  the 
ground.  The  nodes  representing  the  appropriate  Army  C4I  systems  and  the  ATACMS  were  then  added  to 
provide  a  more  robust  cross  section  of  players  for  interaction  with  the  E-8C  and  LGSM  radar  surveillance 
operators. 

Several  components  were  required  to  create  the  ADS-enhanced  operational  environment  used  in  the  ETE 
Test.  In  addition  to  Joint  STARS,  the  ETE  Test  required  a  simulation  capable  of  generating  thousands  of 
entities  representing  the  rear  elements  of  a  threat  force.  For  this  purpose,  the  ETE  Test  team  selected  the 
U.S.  Army’s  Janus  simulation. 

Also,  a  simulation  of  the  Joint  STARS  radar,  called  the  Virtual  Surveillance  Target  Attack  Radar  System 
(V STARS),  that  simulated  both  moving  target  indicator  (MTI)  radar  and  synthetic  aperture  radar  (SAR), 


was  developed  to  insert  the  simulated  targets  into  the  radar  stream  within  the  laboratory  or  on  board  the 
E-8C  while  it  was  flying  a  live  mission. 

The  ETE  Test  consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  that  allowed  the 
mix  of  live  and  simulated  targets  at  an  E-8C  operator’s  console  and  LGSM  operator’s  console.  Phase  2 
evaluated  the  utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a  C4ISR  system  in  a  laboratory 
environment.  Phase  3  moved  components  of  VSTARS  onto  the  E-8C  aircraft,  ensured  that  the 
components  functioned  properly,  and  checked  that  the  synthetic  environment  properly  interacted  with  the 
aircraft  and  the  actual  LGSM.  Phase  4  evaluated  the  ability  to  perform  test  and  evaluation  in  a 
synthetically  enhanced  operational  environment  using  typical  operators. 

ETE  Test  Synthetic  Environment  -  Phase  2  Laboratory  Version 

In  addition  to  the  questions  regarding  the  utility  of  ADS  to  support  DT&E  and  OT&E,  JADS  was  also 
asked  to  determine  if  ADS  had  utility  in  improving  early  operational  assessment  of  major  weapon 
systems.  Phase  2  of  the  ETE  Test  was  designed  to  determine  if  an  ADS-enhanced  test  environment  could 
be  used  for  developmental  testing  and  early  operational  assessment  of  major  weapon  systems. 

During  Phase  2,  the  E-8C  aircraft  was  represented  by  a  laboratory-based  simulation  (VSTARS)  developed 
by  Northrop  Grumman  and  Lockheed  Martin.  VSTARS  is  composed  of  the  radar  software  used  on  board 
the  aircraft  and  several  simulations  representing  aircraft  subsystems.  VSTARS  also  contains  two 
simulations  representing  the  primary  radar  modes  available  from  the  E-8C  aircraft. 

Northrop  Grumman  developed  the  MTI  radar  mode  simulation,  called  MTI  SIM,  by  using  early 
engineering  models  of  the  radar.  The  engineering  models  were  used  to  develop  performance  curves  for 
the  radar.  These  performance  data  were  then  used  to  solve  the  radar  equations  for  a  given  target 
detection.  Included  within  MTI  SIM  was  the  capability  to  adjust  certain  radar  parameters  so  that  the 
simulated  radar  could  be  tuned  to  match  actual  radar  performance. 

Lockheed  Martin  (formerly  Loral  Defense  System-Arizona)  developed  the  SAR  simulation,  called  the 
Advanced  Radar  Imaging  Emulation  System  (ARIES),  based  upon  SAR  simulations  developed  for  the  Air 
Force’s  Wright  Laboratory.  ARIES  was  developed  to  meet  Joint  STARS  specifications  and  its  fidelity  was 
tuned  to  meet  current  performance  fidelity  on  board  the  aircraft.  Northrop  Grumman  integrated  it  into 
VSTARS  and  it  provides  near  real-time  SAR  images  of  anywhere  within  the  radar  area  of  interest. 

VSTARS  provides  the  tester  the  ability  to  conduct  DT&E  of  Joint  STARS  subsystems  other  than  the 
radar.  Its  outputs  are  the  standard  radar  reports  used  within  Joint  STARS,  and  VSTARS  must  run  with 
Joint  STARS  software  normally  used  on  the  aircraft.  During  Phase  2,  Northrop  Grumman  investigated 
the  possible  use  of  VSTARS  to  evaluate  automatic  tracker  software  contained  within  the  next  software 
build  to  be  released  for  Joint  STARS.  The  results  of  this  investigation  will  be  discussed  later  in  this  paper. 

The  development  of  VSTARS  also  provides  the  tester  the  ability  to  evaluate  operational  issues  in  a 
laboratory  environment.  Even  though  Joint  STARS  is  a  fielded  system,  it  is  a  scarce  resource  that  has 
been  heavily  tasked  since  before  its  fielding.  The  lack  of  available  mission  aircraft  and  the  heavy 
workload  placed  on  the  test  aircraft  have  made  the  OT&E  of  systems  associated  with  the  E-8C  extremely 
difficult.  Even  the  Army’s  CGS,  which  is  a  subsystem  of  Joint  STARS,  received  only  a  limited  number  of 
sorties  for  operational  testing. 

During  Phase  2,  the  JADS  ETE  Test  developed  the  previously  described  synthetic  environment  that  could 
be  used  to  support  early  operational  assessment  of  systems,  to  include  Joint  STARS,  and  could  also  be 
used  to  support  OT&E  of  Joint  STARS  and  systems  that  rely  upon  Joint  STARS  for  targeting  information. 
Since  JADS  purpose,  as  stated  earlier,  was  to  investigate  the  utility  of  ADS  for  OT&E  and  not  to  test  Joint 
STARS,  only  one  small  slice  of  the  operational  environment  was  investigated.  The  results  of  our  attempt 
to  use  VSTARS  for  early  operational  assessment  and  OT&E  will  also  be  discussed  later  in  this  paper. 


ETE  Test  Synthetic  Environment  -  Phase  4  Live  Test  Version 


One  of  the  mandated  requirements  for  the  operational  testing  of  C4ISR  systems,  such  as  Joint  STARS,  is 
that  the  actual  system  will  participate  in  the  test.  It  is  not,  nor  has  it  ever  been,  the  intention  of  JADS  to 
replace  live  testing  with  ADS.  Rather,  our  intent  has  been  to  supplement,  or  enhance,  the  live  testing  with 
an  ADS-enhanced  test  environment  that  more  closely  represents  the  actual  operational  environment  the 
system  will  experience  when  operationally  deployed. 

The  Phase  4  synthetic  environment  consisted  of  the  same  synthetic  environment  as  was  used  during  Phase 
2  with  the  exception  that  an  ADS-enhanced  E-8C  was  used.  The  radar  simulations  used  in  VSTARS  were 
moved  to  the  aircraft  and  operated  concurrently  with  the  onboard  radar.  The  data  on  the  10,000  virtual 
targets  were  sent  to  the  aircraft  using  satellite  communications  (SATCOM).  This  enabled  the  aircraft, 
while  flying  over  Fort  Hood,  Texas,  to  observe  live,  mixed  and  virtual  MTI  radar.  The  operators  could 
also  observe  virtual  and  real  SARS.  These  radar  reports  were  then  transmitted  to  the  ground  and  received 
by  an  actual  light  ground  station  module  and  common  ground  station.  The  sensor-shooter-sensor  loop 
was  then  followed  with  the  results  of  the  ATACMS  missile  strikes  being  observed  (survivors  displacing 
and  hulks  remaining)  by  the  surveillance  operators  and  intelligence  analysts. 

The  operational  testing  conducted  during  Phase  4  consisted  of  three  live  trials  using  an  ADS-enhanced 
test  environment  and  five  laboratory  trials  using  the  Phase  2  synthetic  environment.  All  trials  were  more 
than  five  hours  in  duration  with  the  exception  of  the  last  live  trial  which  was  curtailed  because  of  the 
nonavailability  of  tanker  aircraft.  The  lab  trials  filled  in  the  test  scenarios  that  we  were  not  able  to  execute 
because  of  the  lack  of  live  trials  and  repeated,  for  comparison  purposes,  the  scenarios  executed  during  the 
live  trials.  Subsequent  comparison  of  the  live  trials  and  the  lab  trials  revealed  no  difference  in  the 
surveillance  operators  and  intelligence  analysts  actions  with  regard  to  the  virtual  targets.  During  the 
second  live  trial,  instrumented  vehicles  and  radar  reflectors  were  placed  at  Fort  Hood,  and  JADS  verified 
that  the  actual  radar’s  performance  was  not  affected  by  the  ADS-enhanced  test  environment.  The  results 
of  the  Phase  4  operational  test  will  be  discussed  later  in  this  paper. 

Detailed  information  on  the  ETE  Test  is  available  in  an  interim  report  for  each  phase  and  can  be  found  at 
http://www.jads.abq.com.  (After  1  March  2000,  refer  requests  to  HQ  AFOTEC/HO,  8500  Gibson  Blvd. 
SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558  or  SAIC  Technical  Library,  2001  North 
Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

Verification  and  Validation  of  the  Synthetic  Environment 

Both  the  Phase  2  laboratory  version  and  the  Phase  4  live  test  version  of  the  synthetic  environment, 
including  all  elements  of  the  environments,  were  verified,  validated  and  accredited.  The  details  are  too 
extensive  to  cover  in  this  paper,  however  they  are  available  at  the  previously  mentioned  Web  site.  Two 
issues  are  worth  mentioning  and  will  be  discussed  as  a  part  of  this  paper. 

Early  in  the  development  of  the  ETE  Test,  the  issue  of  how  to  validate  VSTARS  was  brought  up. 
Validation  is  the  determination  of  “...the  degree  to  which  a  model  or  simulation  is  an  accurate 
representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model  or  simulation.” 
(DoD  5000. 59-M)  VSTARS  needed  to  represent  the  actual  radar  found  in  the  Joint  STARS  system 
closely  enough  so  that  all  the  operator’s  actions  would  be  identical  to  those  used  for  the  actual  system. 

The  ETE  Test  verification  and  validation  (V&V)  team  decided  to  use  a  validation  procedure  called  a 
Turing  test,  modified  to  meet  our  particular  needs.  A  Turing  test  is  based  upon  experts’  knowledge  about 
the  system  of  interest.  The  experts  are  normally  presented  with  two  sets  of  output  data,  one  obtained  from 
the  simulation  and  one  from  the  system,  and  are  asked  to  differentiate  between  the  two.  If  they  succeed, 
they  are  asked  to  describe  the  differences. 


All  the  experts  knew  that  it  was  not  possible  to  do  what  we  were  doing  without  using  simulation. 
Therefore,  we  presented  them  with  output  data  from  the  simulation  and  asked  them  to  differentiate 
between  the  simulation  and  the  system  based  upon  their  experience  with  the  system.  The  experts  were 
given  unlimited  freedom  to  query  and  interact  with  the  system  for  two  hours.  They  were  provided  no 
instruction  on  the  simulation  except  that  they  were  to  treat  it  as  if  it  was  a  Joint  STARS. 

The  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center  (TRAC),  White  Sands  Missile  Range 
(WSMR),  New  Mexico,  training  directorate,  conducted  the  modified  Turing  test.  Individuals  with  Joint 
STARS  experience  performed  a  two-hour  mission  using  VSTARS  and  then  were  asked  to  compare 
VSTARS  with  Joint  STARS.  The  test  was  conducted  at  the  Northrop  Grumman  facilities  in  Melbourne, 
Florida.  Twelve  individuals  served  as  participants  in  the  test.  Study  participants  represented  the  Joint 
STARS  development,  testing,  training,  and  operations  communities  and  included  Air  Force,  Army,  and 
civilian  personnel.  Personnel  acting  as  operators  worked  at  the  advanced  tactical  work  stations  (ATWS) 
in  the  Northrop  Grumman  Operations  and  Control  Test  Laboratory.  The  system  management  officer 
(SMO)  operated  from  a  workstation  located  in  the  Northrop  Grumman  JADS  Laboratory. 

During  a  pretest  planning  session,  personnel  from  the  Joint  STARS  Joint  Test  Force  briefed  participants 
regarding  the  test  objective,  mission  scenario,  and  mission  objectives.  The  mission  scenario  and  mission 
objectives  were  available  to  provide  structure  to  the  test  if  the  expert  desired  to  use  them.  At  the 
completion  of  each  test  session,  a  TRAC- WSMR  analyst  interviewed  the  two  operators  and  the  SMO  as  a 
group.  The  interview  questions  addressed  a  number  of  issues,  including  how  well  VSTARS  simulated 
Joint  STARS  in  the  moving  target  indicator  and  synthetic  aperture  radar  modes,  and  whether  the 
operators  were  able  to  perfonn  their  assigned  mission.  The  TRAC-WSMR  analyst  also  observed  the 
operators  during  the  test  sessions  and  recorded  operator  comments  pertaining  to  the  simulation. 

All  the  participants  thought  VSTARS  did  a  very  good  job  of  simulating  Joint  STARS  in  the  MTI  and  SAR 
modes.  In  addition,  the  operators  experienced  no  difficulties  in  performing  their  mission  on  the  ATWS. 
In  support  of  this  statement,  operators  noted  that  all  functions  worked  like  Joint  STARS,  the  windows 
were  identical  to  Joint  STARS,  and  the  response  rate  was  the  same  as  that  of  Joint  STARS.  The 
participants  were  very  impressed  with  the  ability  of  VSTARS  to  integrate  the  operational  scenarios  into 
the  radar  products. 

This  ability  to  link  the  operational  scenario,  representing  an  Iraqi  Army  rear  area,  with  the  radar  products 
was  also  commented  on  throughout  the  test.  Veterans  of  actual  Joint  STARS  deployments  stated  that  it 
was  extremely  realistic.  One  of  the  common  complaints  voiced  by  surveillance  operators  and  intelligence 
analysts  was  that  there  was  too  much  operationally  significant  data  to  easily  handle  and  process  during  the 
mission. 

The  full  report  on  the  modified  Turing  test  is  contained  within  the  ETE  Test  V&V  reports  available  on  the 
JADS  Web  site. 

The  second  issue  was  that  we  were  unable  to  completely  verify  and  validate  the  Phase  4  synthetic 
environment  until  the  actual  test  trials.  This  was  because  the  test  synthetic  environment  did  not  exist  until 
the  aircraft  was  orbiting  over  Fort  Hood. 

Extensive  V&V  was  conducted  using  the  test  aircraft  on  the  ground,  and  there  was  a  high  degree  of 
confidence  that  all  would  work  during  the  flight.  However,  the  radar  could  not  be  used  on  the  ground  and 
there  were  major  differences  in  the  SATCOM  link,  such  as  antenna  orientation  during  turns,  that  could 
only  be  investigated  during  an  actual  flight. 

During  the  flights,  data  were  collected  on  the  SATCOM  performance,  and  the  actual  radar’s  performance 
was  measured.  Many  of  the  V&V  steps  for  VSTARS  and  the  synthetic  environment  were  repeated  to 
ensure  both  were  performing  as  expected  under  the  actual  test  conditions. 


End-T6-End  Test  Results  and  Lessons  Learned 


The  ETE  Test  had  the  following  objectives: 

JADS  Issue  1.  What  is  the  present  utility  of  ADS,  including  DIS,  for  T&E? 

JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including  DIS,  during  test 
execution. 

JADS  Objective  1-2.  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E. 

JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E? 
JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS  performance  for  T&E. 
JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support  systems  for  T&E. 
JADS  Objective  2-3.  Develop  and  assess  methodologies  associated  with  ADS  for  T&E. 

The  results  of  the  ETE  Test  are  discussed  in  great  detail  in  the  interim  reports  on  the  four  phases  and  in 
less  detail  in  The  Utility  of  ADS  for  C4ISR  System  Testing  report,  which  can  be  found  on 
www.jads.abq.com.  For  the  purpose  of  this  paper,  we  will  consider  the  results  as  they  apply  to  OT&E,  to 
include  early  operational  assessment,  and  to  DT&E. 

OT&E  and  Early  Operational  Assessment 

The  ETE  Test  determined  that  valid  system  under  test  (SUT)  data  could  be  collected  on  a  complex  C4ISR 
system,  such  as  Joint  STARS,  both  in  a  laboratory-based  ADS  test  environment  and  in  a  live  ADS- 
enhanced  test  environment.  The  Phase  2  ADS  configuration  could  evaluate  15  out  of  45  effectiveness 
measures  of  performance  (MOP)  (including  two  MOPs  not  evaluated  during  the  Joint  STARS  contingency 
operations  test  and  evaluation)  and  all  eight  effectiveness  measures  of  effectiveness  (MOE).  Further,  the 
Phase  2  ADS  configuration  could  be  used  to  evaluate  8  out  of  27  GSM  suitability  MOPs. 

The  Phase  4  ADS  configuration  evaluated  the  same  measures  as  the  Phase  2  ADS  configuration  using  the 
actual  components  of  the  system.  However,  if  additional  elements  (simulated  or  real)  were  added  to  the 
Phase  4  ADS  configuration,  it  could  evaluate  all  45  effectiveness  MOPs  (including  two  MOPs  not 
evaluated  during  the  contingency  operations)  and  all  8  effectiveness  MOEs.  Furthermore,  the  augmented 
Phase  4  ADS  configuration  could  be  used  to  evaluate  the  GSM  and  E-8C  suitability  MOPs  (27  out  of  27 
suitability  MOPs),  all  of  the  human  factors  MOPs,  and  all  of  the  software  MOPs. 

ADS  can  provide  a  useful  tool  for  the  tester  in  support  of  early  operational  testing.  A  laboratory  version  of 
the  SUT  that  performs  to  specifications  can  be  used,  as  was  the  case  during  the  Phase  2  test,  to  assess 
early  operational  effectiveness.  If  the  performance  of  the  simulation  can  be  adjusted,  studies  can  be 
conducted  to  determine  which  performance  values  have  the  greatest  impact  on  the  operational 
effectiveness  of  the  system. 

DT&E 


One  of  the  software  tools  available  to  the  operators  on  board  the  E-8C  is  an  automatic  tracker  (A-tracker). 
This  tool  will  automatically  track  a  set  of  targets  designated  by  the  operator  based  upon  radar  reports  and 
provide  the  operator  with  information  about  the  targets.  Northrop  Grumman  agreed  to  investigate 
whether  or  not  VSTARS  would  be  useful  in  testing  the  tracker  in  the  laboratory  prior  to  live  testing.  Prior 
to  the  use  of  VSTARS,  laboratory  (developmental)  testing  could  only  be  done  on  a  very  limited  basis 
because  of  the  requirement  for  actual  radar  reports. 

Live  developmental  testing  was  also  limited  because  of  limited  resources  and  cost.  In  addition,  when  live 
testing  was  conducted,  it  used  only  a  limited  number  of  test  cases  because  of  the  lack  of  test  assets,  time, 
and  safety  considerations. 


The  version  of  VSTARS  used  by  Northrop  Grumman  to  evaluate  its  DT&E  utility  was  an  early  version 
and  lacked  critical  instrumentation  required  for  the  evaluation.  In  addition,  it  was  determined  that  a 
higher  resolution  simulation  was  required  to  drive  the  entities  involved  in  the  test.  Despite  the 
shortcomings,  the  experiment  indicated  that  VSTARS  was  capable  of  supporting  the  DT&E  of  Joint 
STARS  subsystems,  such  as  the  A-tracker,  provided  the  instrumentation  was  added  and  a  different 
scenario  driver  was  used. 

Several  observations  on  the  utility  of  ADS  to  support  DT&E  were  made. 

-  Multiple  repetitions  of  the  same  scenario  can  be  performed  without  competing  for  scarce  lab 
resources,  test  aircraft,  and  range  resources. 

-  The  use  of  ground-based  simulations  and  hardware  offers  the  potential  for  enormous  cost 
savings,  as  compared  to  a  live  test  with  the  aircraft. 

-  Any  conceivable  test  case  can  be  “flown”  in  the  laboratory  without  worrying  about  safety  or 
limited  assets  provided  the  appropriate  scenario  generator  is  available. 

-  Bad  software  can  be  quickly  discarded  and  new  software  could  be  tried  the  next  day  against  the 
entire  suite  of  test  cases. 

-  Most  importantly,  when  a  live  test  is  flown,  as  it  must  be,  the  testers  can  be  reasonably  sure  that 
they  will  get  the  maximum  value  from  the  flight  and  test  conditions. 

It  must  be  noted  that  VSTARS  ran  on  one  workstation  during  this  test  as  opposed  to  the  extensive  radar 
laboratory  normally  required  for  developmental  testing  of  the  tracker.  It  truly  provided  the  ability  to  do 
desktop  testing.  Use  of  the  Phase  4  test  environment  would  also  allow  the  tester  to  test  against  available 
live  test  cases  and  against  normally  unavailable  virtual  test  cases,  thereby  maximizing  the  use  of  the  test 
flight. 

Lessons  Learned 


-  An  ADS-enhanced  live  test  environment  using  validated  simulations  can  provide  more  realistic 
threats  and  force  levels  than  those  offered  by  conventional  tests,  i.e.,  threats/levels  otherwise 
unavailable  because  of  cost  restrictions,  unobtainable  threats,  etc. 

-  C4ISR  system  testers  can  tailor  the  simulation  entities  operating  in  the  ADS  environment  to 
closely  reflect  the  forces  expected  in  operational  theaters,  thus  further  increasing  the  relevance  of 
the  collected  test  data. 

~  ADS  allows  testers  to  have  more  control  over  the  specific  aspects  of  the  scenarios  of  interest  and 

to  expand  their  test  concept  and  design.  A  typical  constraint  to  test  concept  development  is  the 
number  and  types  of  units  readily  available  for  a  test. 

-  ADS  technology  can  easily  eliminate  the  conventional  testing  disadvantage  of  insufficient 
battlespace.  For  example,  the  National  Training  Center  occupies  about  one  thousand  square 
miles.  In  contrast,  the  battlespace  for  Phase  2  of  the  ETE  Test  was  almost  ten  times  larger.  In 
fact,  ADS  technology  is  capable  of  supporting  even  larger  battlespaces. 

-  Using  ADS,  testers  can  conduct  more  test  trials  for  longer  periods  of  time.  During  the  Phase  2 
test,  the  ETE  Test  team  was  able  to  conduct  five  test  trials,  lasting  six  hours  each,  within  a  5-day 
period.  A  maximum  of  three  trials  could  have  been  conducted  using  the  test  aircraft.  If 
additional  shifts  of  operators  were  available,  the  test  trails  could  have  lasted  longer  at  little 
additional  cost.  Because  of  scheduling  issues,  crew  rest,  and  the  lack  of  test  aircraft,  it  would  be 
difficult  to  extend  a  live  test  trial  beyond  five  to  six  hours  on  station.  During  the  Phase  4  test,  the 
ETE  Test  team  was  able  to  conduct  eight  test  trials  using  three  live  test  trials  and  five  laboratory 


test  trials.  The  laboratory  test  trials  were  mainly  conducted  during  the  periods  between  live  test 
trials  at  little  or  no  additional  cost. 

-  The  ETE  Test  also  demonstrated  that  there  are  no  real  technical  barriers  to  using  an  ADS 
environment  to  provide  a  realistic  test  environment  for  a  C4ISR  system.  This  is  due  to  the  high 
reliability  of  the  network  architecture  underlying  an  ADS  environment  and  the  dramatic 
increases  in  computer  processing  and  storage  capabilities  over  the  past  few  years.  Rather,  the  key 
constraints  to  ADS  testing  are  the  familiar  ones  of  time  and  money. 

-  ADS  allows  the  tester  to  assemble,  using  models,  simulations,  emulators,  and  fielded  systems, 
systems  of  systems  that  either  have  not  yet  been  built  or  would  not  exist  except  in  time  of  war. 
During  the  ETE  Test,  an  ATACMS  missile  was  fired  at  an  enemy  force  based  on  operationally 
realistic  intelligence  collected  by  Joint  STARS  and  processed  by  an  element  of  ASAS.  This  was 
done  using  fielded  C4ISR  systems  that  were  electronically  linked  and  functioning  as  they  would 
in  combat.  Army  intelligence  subject  matter  experts  stated  that,  to  the  best  of  their  knowledge, 
this  had  never  been  done  before  the  ETE  Test. 

-  ADS  can  easily  provide  models  and  simulations  that  represent  the  threats  of  today  and  the  threats 
of  tomorrow.  All  models  and  simulations  of  threats,  no  matter  what  their  fidelity,  are  data 
driven.  The  fidelity  is  determined  by  the  accuracy  and  detail  contained  within  the  data  and  the 
fidelity  of  the  algorithms  that  use  the  data  to  depict  the  threat.  During  the  ETE  Test,  the  threats 
used  were  those  currently  fielded  within  the  Iraqi  Army.  They  could  have  just  as  easily  been  low 
observable  threats  from  a  future  battlefield. 

Conclusion 


As  stated  in  the  JADS  report,  The  Utility  of  ADS  for  C4ISR  System  Testing : 

Since  all  C4ISR  systems  contain  the  same  basic  elements  (e.g.,  sensor(s),  sensor 
platform(s),  command  and  control  elements,  communication  lines,  and  computer 
hardware  and  software),  the  extension  of  the  ETE  Test  results  to  other  possible  C4ISR 
applications  is  relatively  straightforward.  ADS  technology  allows  the  evaluation  of 
human-in-the-loop  actions,  decision  processes,  timelines,  and  interoperability  which 
digital  simulations  do  not  model  well.  Using  ADS,  a  mission-level  scenario  model  can 
be  linked  to  actual  C4ISR  hardware  and  software  with  tactical  operators-in-the-loop  and 
tactical  communications  links  for  realistic  testing  in  force-on-force  scenarios  which 
cannot  be  accomplished  in  live  testing. 

ADS  is  not  just  of  value  to  C4ISR  T&E  but  can  be  applied  throughout  the  system 
acquisition  life  cycle.  In  fact,  the  benefits  of  using  ADS  are  best  realized  over  the  life  of 
a  program.  ADS  is  an  enabling  technology  for  Simulation  Based  Acquisition  (SBA)  and 
the  Simulation,  Test,  and  Evaluation  Process  (STEP)  as  applied  to  C4ISR  systems,  since 
these  systems  are  naturally  distributed  by  nature. 

Future  uses  of  ADS  currently  being  discussed  are  a  follow-on  test  of  the  CGS,  the  operational  test  for  the 
Block  20  E-8C  (computer  replacement  program),  and  operational  testing  of  the  Block  II  ATACMS 
missile.  All  these  programs  are  currently  investigating  the  use  of  VSTARS  and  elements  of  the  ETE  Test 
synthetic  environment. 
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